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Image  Segmentation  with  Detection  of  Highlights  and  Inter-reflections  Using  Color 

Ruzena  Bajcsy,  Sang  Wook  Lee  and  Ales  Leonardis* 

Department  of  Computer  and  Information  Science,  University  of  Pennsylvania,  Philadelphia,  PA  19104 

Abstract 

We  present  an  approach  to  the  construction  of  a  computational  model  for  color  image  segmen¬ 
tation  based  on  the  physical  properties  of  sensors,  illumination  lights  and  surface  reflectances. 

Using  the  established  model,  we  perform  color  image  segmentation  and  detect  small  inter¬ 
reflections  as  well  as  highlights. 

1  Introduction 

Color  is  one  of  the  major  features  in  identifying  objects,  and  color  vision  has  been  the  most 
intensively  studied  sensory  process  in  human  vision.  In  machine  vision,  however,  the  use  of  color 
has  not  been-  the  subject  of  active  research,  although  color  has  recently  received  attention  as  a 
useful  property  for  image  segmentation  and  object  recognition.  The  purpose  of  color  vision  is  to 
extract  aspects  of  the  spectral  property  of  object  surfaces,  while  at  the  same  time  discounting 
various  illuminations  in  order  to  provide  useful  information  in  image  analysis.  The  most  notable 
applications  include  recognition  and  identification  of  colored  objects  and  image  segmentation. 

There  have  been  approaches  to  image  segmentation  based  on  histograms  of  red  (R),  green  (G) 
and  blue  (B)  values  or  on  transformed  values  such  as  YIQ  [10]  [11],  or  by  using  color  edges  in  addition 
to  intensity  edges  [9].  These  algorithms  are  primarily  extentions  of  intensity  segmentation  to 
*Also  at  Fakulteta  za  elektrotehniko  v  Ljubljani,  University  of  Ljubljana,  Yugoslavia, 


1 


multidimensional  processing  with  extra  information  provided  by  color.  Thus  they  cannot  distinguish 
between  different  edge  types  due  to  material  changes,  highlights,  shading,  and  shadow. 

Also,  there  has  been  an  approach  to  detection  of  highlights  by  combining  the  traditional  segmen¬ 
tation  with  physics-based  post-processing  [4]\  Since  traditional  color  image  segmentation  algorithms 
generally  have  not  accounted  for  the  influence  of  optical  effects  on  object  colors,  the  method  suffers 
from  many  erroneous  regions.  If  the  physical  models  were  included  at  the  segmentation  stage, 
many  of  these  regions  could  be  properly  classified.  The  first  physical  approaches  to  color  image 
understanding  have  been  introduced  only  recently  [1]  [2].  Detection  of  highlights  is  acheived  by 

j 

clustering  color  points  in  RGB  space  using  the  dichromatic  property  of  dielectric  materials.  Since 
the  spectral  curves  of  the  camera  and  filters  are  not  included  in  these  models,  the  metric  in  the 
RGB  space  varies  with  the  choice  of  the  sensors. 

In  addition  to  highlights,  shading  and  shadow,  there  are  other  impediments  to  observing  real 
material  changes  such  as  variation  of  illumination  and  multiple  reflections  between  objects.  Objects 
receive  not  only  the  lights  directly  from  illumination  sources  but  also  the  reflected  lights  from  other 
objects.  The  latter  induces  a  change  of  the  perceived  surface  color  which  we  call  inter-reflection. 
The  process  of  discounting  illuminations  from  the  surface  reflectance,  namely  color  constancy,  is  a 
heavily  studied  subject  in  color  vision  [8]  [6]  [5],  though  successful  applications  are  yet  limited. 

In  this  paper  we  present  an  approach  to  the  construction  of  a  computational  model  for  color 
segmentation,  not  only  with  detection  of  highlights  but  with  detection  of  small  color  changes  induced 
by  inter-reflections  as  well.  We  use  the  dichromatic  model  [1]  for  dielectric  materials  and  develop  a 
metric  space  based  on  the  physical  properties  of  sensors  in  order  to  better  manipulate  light-surface 
interactions. 

We  begin  in  Section  2  with  sensing  and  metric  representation  of  color  in  a  machine  vision  system. 
In  the  following  section,  we  describe  a  method  of  color  constancy  using  a  reference  plate.  In  Section  4 
a  surface  reflectance  model  is  established  from  various  mechanisms  of  surface  reflections.  Various 
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optical  phenomena  such  as  shading,  shadow,  highlights  and  inter-reflections  between  objects  are 
explained  in  Section  5,  in  terms  of  the  established  surface  reflectance  model.  In  Section  6,  we 
propose  a  method  of  segmentation  and  highlights  and  inter-reflection  detection  between  objects. 
We  present  some  of  our  initial  results  in  Section  7,  and  suggestions  for  further  studies  are  given  in 
the  following  section. 

2  Representation  and  Sensing  of  Color 

Representation  of  color  with  finite-dimensional  linear  models  has  been  a  topic  of  many  studies 
[12]  [13]  [7]  [14].  There  have  been  some  approaches  to  obtaining  characteristic  basis  functions  by 
investigating  many  samples  of  daylight  and  surface  reflectances  [13]  [12].  It  has  been  suggested  that 
although  the  number  of  basis  functions  required  to  completely  describe  full  spectra  is  essentially 
infinite,  a  small  number  of  basis  functions  can  provide  good  spectral  approximations  of  most  natural 
illuminants  and  surface  reflectances.  Our  approach  is  to  represent  measured  colors  in  a  metric  space 
in  order  to  better  manipulate  physical  properties  of  surface  colors  for  image  segmentation. 

Both  for  surface  reflectance  and  for  illumination,  we  here  use  the  first  three  Fourier  bases 
for  their  mathematical  simplicity  and  for  their  sufficient  effectiveness  in  resolving  natural  colors. 
Within  our  framework  developed  here,  however,  better  models  of  basis  functions  (such  as  the  first 
three  of  Cohen’s  reflectance  basis  functions  [12]  and  those  of  Judd’s  illupiinant  basis  functions  [13]) 
can  be  used. 

For  the  three-dimensional  linear  model,  surface  reflectance  and  illumination  can  be  represented 
as  a  weighted  sum  of  basis  functions  expressed  as: 

2 

S(x,y,\)  =  J2<Ti(x>y)Si(  A)>  (1) 

»'=o 

2 

E(x,y,X)  =  Y^£j(x,y)Ej(X),  (2) 

3- 0 
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BASIS_FUNCTIONS 


Figure  1:  Fourier  basis  functions 

where  S{( A)  and  Ej(X)  are  the  basis  functions,  and  crj(x,y)  and  ej(x,  y)  are  the  scalar  weighting 
factors  at  ( x,y ).  The  basis  functions 


So(\)  =  1)  .?i(A)  —  sinX,  $2( A)  =  cosA  (3) 


are  shown  in  Figure  1.  • 

The  color  image  sensing  is  performed  with  a  CCD  camera  using  filters  of  different  spectral 

responses.  The  measured  color  signal  I(x,y,  A)  is  obtained  as  a  product  of  the  spectral  power 

distribution  ( SPD )  of  illumination  E(x,y,  A)  and  the  spectral  response  of  the  surface  reflectance 

) 

function  !?(£,  y,  A),  i.e., 

I(x,y,X)  =  E(x,y,\)  ■  S(x,y,  A).  (4) 

With  3  filters  (usually  R,  G  and  B),  the  quantum  catch  or  the  measured  signal  from  the  camera  is 
given  by 

Pk{x,y)=  [  I(x,y,X)  ■  Qk(\)d\,  (5) 

E  i 

where  Q *(A)  and  pk{x,y)  for  k  =  0,  1,  2  are  the  spectral  response  of  the  k— th  filter  and  the  camera 
output  through  the  k- th  filter  at  ( x,y ),  respectively.  The  wavelengths  Aj  =  400  nm  and  A2  = 
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Figure  2:  (a)Spectral  responses  of  3  filters,  (b)  Spectral  response  of  CCD  and  IR  filter 

700  nm  cover  the  range  of  the  visible  spectrum.  Figure  2  (a)  shows  the  spectral  response  of  the  3 
filters:  Qo(A),  <3i(A)  and  CMA).  The  curves  are  obtained  from  filter  measurements  with  a  spectro¬ 
photometer  and  are  multiplied  by  the  product  of  the  spectral  sensitivity  of  the  CCD  camera  and 
the  infra-red  cut-off  (IR)  filter  which  is  shown  in  Figure  2  (b).  The  filters  integrate  the  spectral 
responses  over  the  given  range.,  and  consequently  act  as  an  anti-aliasing  filter  in  sampling  the  color 
signal. 

With  3  basis  functions,  the  relationship  between  the  sensor  response,  illumination  and  re¬ 
flectance  is  given  as 

p  —  —  fiV  (6) 

where  the  elements  of  and  Q5  in  the  k- th  row  and  j- th  (or  i- th)  column  are  respectively 

OTkj=  C  j^aiSiEjQkdX,  (7) 

**x\  x— 0 

C?  EjSiQkdX,  (8) 

-i  i— o 
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3  Color  Constancy 


To  obtain  the  surface  reflectance  of  objects,  which  is  the  primary  information  for  image  analysis, 
the  influence  of  illumination  should  be  discounted  from  the  measured  color  in  the  image.  This 
process  is  called  color  constancy.  While  the  general  solution  for  color  constancy  with  unknown 
illumination  demands  complex  algorithms  and  some  spectral  and/or  spatial  constraints  [15]  [6],  it 
is  simple  to  remove  the  known  illumination  measured  with  a  reference  object.  One  assumption  is 
that  the  objects  of  interest  are  illuminated  by  light  sources  of  the  same  SPD ,  and  the  reference 
object  is  applicable.  If  we  use  a  reference  plate  with  known  reflectance  <rre^,  the  SPD  of  illumination 
obtained  from 

eref  =  (o^rv  (9) 

represents  the  spectral  composition  of  the  global  illumination  throughout  the  image  area.  With 
the  normalized  er^JTm^  the  calculation  of 

a  =  .  (10) 


leads  to  the  color  signal  under  whitened  illumination  expressed  as: 


I(x,y,\)  =  e0(x,y) ;  [<To(z,2/)<Sb(A) 

+<?i(x,  y)Si(X)  +  (T2(x,  j/)52(A)]. 

I 

4  Surface  Reflectance  Model 


(ll) 


Once  the  effects  of  color  illumination  have  been  eliminated,  the  color  image  A)  represents 

only  the  surface  reflectance  properties  multiplied  by  the  illumination  intensity  Sq. 

The  analysis  of  the  color  image  demands  proper  models  for  various  optical  phenomena  of  surface 
reflections.  We  use  the  dichromatic  model  for  dielectric  materials  such  as  plastic,  paper  and  paint  [1] 
[3].  The  surface  reflectance  of  dielectric  materials  has  two  different  optical  properties.  Highlights 
are  from  specular  reflection  and  depend  on  the  refractive  indices  of  the  vehicle  materials.  On 
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ther  other  hand,  body  or  diffuse  reflection  occurs  inside  the  material  due  to  the  pigments.  The 
distribution  of  pigments  determines  the  color  of  the  exiting  light  by  selectively  absorbing  certain 
wavelengths. 

The  magnitude  of  the  image  signal  is  also  affected  by  geometrical  factors  such  as  surface  ori¬ 
entation,  illumination  and  viewing  directions  [16].  So  far,  we  have  assumed  that  the  geometrical 
factors  are  implicitly  included  either  in  E(X)  or  in  5(A).  More  precisely,  the  geometrical  weighting 
can  be  regarded  as  the  local  variation  of  illumination  density  on  a  surface  patch.  However,  since 
the  geometrical  weighting  has  different  mechanisms  for  specular  and  body  reflections,  it  would  be 
convenient  to  include  it  in  the  surface  reflectance.  If  we  consider  the  factors  as  part  of  the  surface 
reflectance  of  the  dichromatic  model,  the  surface  reflectance  can  be  modeled  as 


S(  A)  =  gs(G)Ss(  A)  +  gB(G)SB(  A),  (12) 

where  S$( A)  and  Sg( A)  are  the  specular  and  the  body  reflectance,  respectively;  g$  and  g&  are  the 
geometrical  factors  for  the  specular  and  the  body  reflection,  respectively;  and  G  is  .the  geometrical 
variable  that  accounts  for  the  viewer  direction  and  the  source  direction,  where  each  is  relative  to 
the  surface  normal.  Now  5s(A)  and  Sb( A)  represent  the  real  surface  reflectance  of  object,  i.  e., 
albedos .  For  most  vehicle  materials,  the  refractive  index  is  weakly  dependent  on  A.  Thus,  Ss{ A)  is 
almost  constant  over  the  visible  range  of  light,  while  the  body  color  is  determined  by  Sb( A). 

i 

Given  that  the  specular  reflection  is  flat  over  the  spectrum,  we  can  rewrite  the  color  image 
signal  as 

I  =  (£oEq SiEi  €2^2)' 

(13) 

[(j/so'os  +  9b^ob)So  +  9bc\sS\  +  ob^b^t], 

where 

Co  =  9SV0S  +  9B&0B,  -  gB&lB,  02  =  9B &2B-  (14) 

By  using  color  constancy,  we  can  remove  the  terms  £\Ex  and 
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(a)  (b) 

Figure  3:  (a)Color  clusters  on  S  space,  and  (b)  Spatial  variation  of  intensity  due  to  shading  and  highlight 

5  Image  Analysis  on  (Sq,  Si,  S2) 


For  identification  of  the  surface  reflection  mechanisms  explained  above,  we  represent  the  color  on 
the  orthogonal  3-dimensional  S  space.  The  S  space  is  similar  to  the  opponent  space  which  is 
convenient  for  describing  intensity,  hue  and  saturation.  The  definitions  of  intensity,  saturation  and 
hue  for  our  work  are  defined  as: 


intensity 

—  <?o, 

hue 

—  arctan  ^L 

saturation 

II 

0  [£_ 

where  |<Ji2 


5,1  Shading  and  Shadow 


(15) 


Shading  on  surfaces  of  uniform  color  is  due  to  variations  in  geometry  G  in  gs( G).  As  can  be  seen 
in  Equation  (13),  the  shading  can  be  modeled  as  the  weighting  of  <Jqb->  a\B  and  &2B  by  the  scaling 
factor  gs-  This  corresponds  to  a  linear  cluster  of  color  points  through  the  origin  in  S  space,  as 
shown  in  Figure  3  (a).  Shadows  are  due  to  the  screening  of  illumination  by  other  objects,  which 
can  be  interpreted  as  the  variation  of  £p,  £j,  and  e 2  for  a  constant  factor.  Thus  the  shadow  does 
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not  change  the  shape  and  the  orientation  of  the  linear  cluster,  but  moves  the  color  points  toward 
the  origin  along  the  line. 

Since  the  saturation  is  the  slope  of  this  line  with  respect  to  the  5o-axis,  the  saturation  values  of 
the  color  points  on  the  linear  cluster  are  constant.  The  slope  of  the  line  can  be  changed  by  colored 
illumination. 

5.2  Highlight 

Since  the  spectral  response  of  the  specular  reflection  is  flat  for  dielectrics,  the  illuminating  light 

* 

is  reflected  from  the  surface  without  any  change  of  color.  Under  white  illumination  (or  whitened 
illumination  by  color  constancy),  highlights  add  whiteness  (tfos)  to  the  body  colors  by  g$( G). 
As  the  result,  the  color  points  in  S  space  are  shifted  upward  along  the  So  direction  due  to  the 
highlights.  In  general,  the  distribution  of  highlights  can  spatially  vary  over  the  wide  area  of  shaded 
matte  surface  as  shown  in  Figure  3  (b),  so  that  they  form  planar  clusters  which  include  the  linear 
clusters  made" by  shading  and  shadow  on  the  S  space  as  shown  in  Figure  3  (a).  The  planes  are 
perpendicular  to  the  S\  —  S2  plane  under  white  illumination  or  whitened  illumination  by  color 
constancy. 

5.3  Inter-reflection 

1 

So  far  we  have  been  concerned  only  about  the  first  reflection  of  light.  As  there  are  many  objects, 
the  object  surface  of  interest  receives  not  only  the  light  from  the  illumination  sources  but  also 
the  reflected  light  from  the  other  objects.  The  latter  causes  the  local  change  of  illumination,  as 
illustrated  in  Figure  4.  The  change  of  surface  color  due  to  the  multiple  reflections  between  objects 
is  named  inter-reflection. 

The  local  influence  of  the  reflected  light  from  other  objects  is  in  fact  the  local  variation  of 
illumination.  Suppose  that  we  achieve  perfect  color  constancy.  In  this  case  the  local  influence  of  the 
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global  illumination 


Figure  4:  Inter-reflection 

reflected  light  from  other  objects  can  be  discounted  together  with  the  global  ilumination.  However, 
since  we  use  color  constancy  for  discounting  only  the  global  illumination,  the  inter- reflection  remains 
as  a  local  change  of  surface  reflectance.  While  large  inter-reflection  results  in  a  completely  different 
cluster  of  points,  small  inter-reflection  can  be  identified  as  a  small  deviation  from  the  linear  cluster 
as  shown  in  Figure  3  (a).  In  general,  the  color  points  of  inter-reflections  do  not  fall  in  the  planar 
region  formed  by  highlights  and  body  reflections,  since  inter-reflection  can  change  the  perceived 
hue  as  well  as  saturation  of  the  surface.  In  Figure  4  the  specularly  reflected  lights  from  the  objects 
B  have  the  same  SPD's  as  the  original  illumination  since  the  specular  reflectance  of  dielectrics  is 
spectrally  flat..  Thus  the  specular  inter-reflections  cannot  be  distingushed  from  the  highlights.  On 
the  other  hand,  the  body  reflection  from  the  object  B  produces  a  local  illumination  on  the  object 
A,  inducing  a  change  of  the  perceived  surface  color  of  the  object  A. 


6  Image  Segmentation 

Our  segmentation  method  is  based  on  the  assumption  that  the  image  consists  of  patches  of  object 
surfaces  which  have  uniform  color,  i.e.,  the  image  can  be  divided  into  many  regions  of  uniform  hue 
and  saturation  regardless  of  surface  structure  and  finishing. 
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S* 


,  Figure  5:  Color  clusters  in  S  space  (top  view) 

One  of  the  advantages  of  having  the  hue  information  in  addition  to  the  image  intensity  is 
that  the  hue  of  a  surface  is  more  stable  than  intensity  values  under  changes  in  the  geometrical 
relationship  between  the  surface  normal  and  the  illumination  direction.  Hue  segmentation  can  be 
done  by  hue-histograming  or  region-splitting  based  on  hue  values. 

Under  white  illumination  (or  whitened  illumination  by  color  constancy),  the  planes  formed 
by  highlights  and  body  reflections  are  perpendicular  to  the  S\  —  S?  plane  as  shown  in  Figure  5. 
Because  the  highlights  are  in  the  same  plane  as  the  body  color,  the  specular  reflection  and  the  body 
reflection  from  the  same  object  are  segmented  together  by  the  hue  segmentation.  Without  color 
constancy,  the  color  cluster  due  to  specular  reflection  is  generally  not  in  parallel  with  the  So  axis, 
but  rather  lies  parallel  to  and  points  in  the  same  direction  as  the  illumination. 

Further  segmentation  and  detection  of  highlights  and  inter-reflections  can  be  achieved  using 
saturation  values.  Figure  6  (a)  and  (b)  illustrate  the  color  clusters  in  the  So  —  saturation  plane 
within  a  given  range  of  hue  values  after  hue  segmentation,  and  in  the  projected  Si/S0  —  S2/S0 
plane,  respectively.  As  shown  in  the  figures,  since  shading  and  shadow  scale  op,  and  op  by 
the  same  factor,  they  do  not  change  the  saturation  value.  Thus,  an  object  region  with  a  given 
color  has  a  constant  saturation  value.  On  the  other  hand,  highlights  increase  only  <tq  under  white 
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(a)  (b) 

Figure  6:  (a)  Intensity-saturation  plane,  and  (b)  Hue-saturation  plane 

illumination,  thereby  always  decreasing  the  saturation.  The  inter-reflections  may  affect  oo,  <J\  and 
<J2,  which  implies  that  hue  and  saturation  change.  Although  inter-reflections  can  possibly  increase 
the  saturation  or  change  the  hue  values  of  the  surface  without  affecting  the  saturation  value,  in 
most  cases  they  decrease  the  saturation.  This  is  because  the  neighboring  objects  usually  decrease 
the  purity  of  the  surface  color  both  by  specular  and  body  reflection.  Therefore  application  of  local 
thresholding  on  saturation  values  enables  us  to  separate  the  color  clusters  of  highlights  and  most 
of  the  small  inter-reflections  from  those  of  body  reflections. 

7  Experimental  Results 

In  order  to  test  the  algorithm,  we  perform  an  experiment  on  plastic  color  balls.  Four  tungsten 
lights  are  used  as  illumination  sources  in  two  directions  (right  and  left  sides).  They  are  spatially 
diffused  in  order  to  reduce  the  contrast  between  highlights  and  matte  colors.  All  the  measurements 
are  in  the  linear  range  of  our  8-bit  image  digitizer,  and  the  camera  output  is  linearized  by  setting 
the  gamma  correction  factor  of  the  CCD  camera  to  1.  Color  constancy  is  performed  using  a  white 
reference  plate  of  known  spectral  reflectance. 


Figure  7  (a),  (b)  shows  the  intensity  (iSo)  and  saturation  images  and  Figure  8  (a)  and  (b)  shows 


the  original  image  projected  on  S i,  and  £2  axes,  after  the  color  constancy. 


Figure  7:  (a)  Intensity  (5o)  (b)  Saturation 
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Figure  8:  (a)  Si  (b)  S2 


Figures  9  and  10  show  the  color  clusters  in  S  space  after  implementing  color  constancy.  As 
expected,  the  color  points  from  highlights  form  planar  clusters  while  shading  makes  linear  clusters. 
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The  deviations  from  the  lines  and/or  planes  are  due  to  the  errors  and  noise  in  measurements  as 
well  as  the  errors  in  modeling  and  nonuniformity  the  surface  color. 
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Figure  9:  (a)  Perspective  view  (b)  Top  view 
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Figure  10:  (a)  Front  view  (b)  Side  view 

Hue  segmentation  is  achieved  using  the  hue-histogram  shown  in  Figure  11.  The  results  are 
partially  shown  in  Figure  12  and  Figure  13.  After  the  hue  segmentation,  application  of  local 
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thresholding  in  each  segment  results  in  the  separation  of  body  reflections  from  highlights  and  inter¬ 
reflections.  Figure  14  (a)  shows  separated  highlights  and  inter-reflections  which  are  the  deviations 
from  the  linear  clusters  of  body  reflections.  Most  of  the  strong  inter-reflections  are  due  to  the 
specular-specular  reflections,  since  the  specular  reflection  is  much  stronger  than  the  body  reflection. 
The  separated  body  reflections  are  shown  in  Figure  14  (b).  Inter- reflections  are  most  apparent 
between  objects  of  different  colors. 
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Figure  11:  Hue  histogram 
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Figure  14:  (a)  Highlights  and  inter-reflections  (b)  Matte 


8  Discussion 

We  use  objects  of  saturated  colors  for  our  initial  experiments  in  order  to  demonstrate  the  efficacy 
of  the  algorithm  under  the  assumption  that  objects  consist  of  surface  patches  of  uniform  hue  and 
saturation  (not  intensity).  The  segmentation  is  performed  only  in  S  space  without  using  any  spatial 
information.  Since  separated  regions  of  similar  color  can  be  segmented  together,  segmentation  of 
more  complex  scenes  requires  partitioning  an  image  in  the  spatial  domain.  Therefore,  the  use  of 

i 

spatial  information  is  desirable  in  order  to  further  improve  the  segmentation. 

As  mentioned  earlier,  the  deviations  of  surface  colors  from  linear  clusters  of  shading  result  from 
any  measurement  errors  and  noise  as  well  as  from  the  non-uniformity  of  pigment  density.  Errors  can 
also  be  generated  in  the  color  constancy  even  with  the  reference  object  due  to  the  finite-dimensional 
representations  of  surface  reflectance  and  illumination.  Another  important  issue  is  the  proper 
determination  of  the  local  thresholds  on  the  saturation  values,  since  this  is  the  key  to  the  successful 
detection  of  highlights  and  inter-reflections.  For  our  sample  image,  we  obtained  the  threshold  values 
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by  examining  the  deviation  in  the  saturation  values  in  uniform  color  regions.  We  found  that  the 
size  of  the  deviations  across  different  parts  of  the  image  is  about  the  same.  Therefore,  the  observed 
deviation  values  can  be  used  for  determining  thresholds  for  any  measurements  from  the  same 
experimental  settings.  The  detection  of  highlights  and  inter-reflections  can  be  further  improved 
by  locally  adaptive  adjustment  of  thresholding.  With  proper  local  thresholds,  the  algorithm  is 
successful  in  detecting  small  highlights  and  inter-reflections  which  cannot  be  easily  clustered  in  S 
space.  Note  that  the  separated  inter-reflections  do  not  represent  the  real  color  of  local  illumination 
since  the  color  change  due  to  the  inter- reflect  ions  is  a  multiplicative  process. 

j 

So  far  we  have  used  a  reference  object  for  color  constancy.  This  was  based  on  the  assumption 
that  global  illumination  sources  have  the  same  SPD.  As  we  have  more  illumination  sources  of 
different  SPD1  s  we  need  general  color  constancy  algorithms  to  account  for  spatial  variation  of 
illumination.  Most  of  the  color  constancy  algorithms  suggested  so  far  require  the  constraint  that 
illumination  varies  smoothly  in  space.  When  the  illumination  sources  are  relatively  close  to  objects 
and  the  objects  are  highly  structured,  the  illumination  also  varies  highly  in  space  depending  on 
the  surface  orientations.  Furthermore  inter-reflections  do  not  appear  smooth  in  general.  Although 
small  inter-reflections  can  be  detected  and  removed  by  observing  the  change  of  surface  color,  large 
inter-reflections  produce  completely  different  clusters  of  color  points  which  can  only  be  separated 
by  color  constancy.  Currently  we  are  developing  a  color  constancy  algorithm  which  can  resolve 

i 

highly  varied  illumination  in  space. 

9  Conclusion 

In  this  technical  report  we  present  an  approach  to  color  image  segmentation  with  the  detection  and 
separation  of  highlights  and  inter-reflections  between  objects.  To  better  represent  and  process  the 
image  color,  a  color  metric  space  is  developed  based  on  the  physical  model  of  the  camera  and  filters. 
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The  measured  color  in  R,  G,  B  space  is  transformed  into  the  metric  space  on  a  set  of  orthogonal 
basis  functions.  With  uncorrelated  orthogonal  values,  we  can  manipulate  each  component  of  color 
separately  or  in  combination.  Since  the  usual  illumination  is  spectrally  colored,  calibration  of  the 
measured  image  is  performed  with  a  white  object  of  reference  to  whiten  the  illumination.  While  the 
spectral  distribution  of  the  object  surface  is  not  changed  by  shading  and  shadow  under  the  white 
illumination,  it  is  affected  by  highlights  and  inter-reflections  between  the  objects.  As  the  highlights 
add  whiteness  to  the  object  color,  they  can  be  detected  by  observing  the  change  in  saturation  of 
the  uniformly  colored  objects.  The  inter-reflections  are  also  detected  with  the  change  in  saturation, 
as  well  as  in  hue  values.  The  experimental  results  show  that  within  the  framework,  our  model  of 
color  image  interpretation  performs  well. 
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